Background: To evaluate the learning outcomes, objective methods of assessment play an important role. With limited resources in medical colleges, it is difficult to implement objective methods like objectively structured practical examination (OSPE). This study is an effort to amalgamate objectivity in conventional type of assessment.

Methods: 33 students of first MBBS were assessed independently by 3 examiners during practical examination. One examiner awarded marks in conventional manner while two other examiners used semi-structured assessment taking into consideration the affective, cognitive and psychomotor domain of the students. Linear mixed effects analysis of the relationship between the outcome score and methods of assessment were performed using statistical software R. Two models were constructed: first, a null-model without any predictor and examiners as random effect and second, a full-model with method of assessment as fixed effects and examiners as random effects.

Results: Intra-class-correlation (ICC) was 0.24 in null model and 0.01 in full model. The decrease in variance due to examiners between the models can be accounted to different methods of assessment. Pearson’s correlation coefficient between the two semi-structured assessments was 0.754 as compared between conventional and semi-structured assessment, which was 0.488 and 0.466.

Conclusion: Linear mixed effects analysis of the relationship between the outcome score and methods of assessment shows the significant effect of types of assessment on the outcome score of the students. Semi-structured assessment is better in terms of validity, reliability, feasibility and applicability.

Method of assessment is an important part of medical curriculum as learning of students depend greatly on that. Assessment methods are analysed objectively in terms of its validity, reliability, feasibility, acceptability & educational impact (1,2). A properly designed assessment method has many benefits. It meets the learning objectives, maintains standard of students, help in giving feedback to students as well as teachers and train the students to deal with real life situations (3).

While considering the practical examination, there are different patterns of assessment that can vary from subjective traditional method to more objective OSPE (objectively structured practical examination) method. Both the patterns have their own merits and limitations although properly conducted OSPE seems to have an edge in terms of objectivity and uniformity (4, 5).

Establishment of department of medical education and imparting training to teachers by various teachers-training programmes have helped in improving the assessment methods (6). It has tried to objectify the assessment of examination and bring uniformity in pattern of examination. At the same time, it is also important to be aware of the resources available and their optimum use with minimal effect on the quality of assessment.

The combination of conventional practical examination and OSPE may improve the validity of the examination (7). With this background, we structured the conventional manner of rewarding marks to the students for an experiment so that an examiner is bound to test the students less subjectively and more objectively. The total marks of an experiment or assignment were categorized into pre-defined 3 or 4 parts giving them appropriate weightage and thus testing most of the skills or domain intended to test. Conventional assessment is then compared with semi-structured assessment method.

In the null-model, ICC was found to be 0.24, which is the proportion of variance in the outcome score that is attributable to examiner level factors. It can be assumed here that part of this variance may be due to the different methods of assessment. For that, we also calculated the ICC from the full-model, which came out to be 0.01. Thus, the decrease in variance due to examiners between the models can be accounted to different methods of assessment.

We also calculated the R - squared from the variance component of both the models and it was found that 96% of the variance was explained by use of different methods of assessment. P-value was also statistically significant (<0.05) for the fixed effect of the methods of assessment.

Pearson’s correlation coefficient between the two semi-structured assessment was 0.754 as compared between conventional and semi-structured assessment which was found to be 0.488 and 0.466.

Conventional subjective pattern of assessment is in place since a very long time. It generally depends on the experience and knowledge of examiners. While on one hand, it may not test all the objective traits or skills (cognitive, psychomotor, affect, behavioural etc.) of the students, on the other hand, it may give an undue weightage to one or the other skill. It can be very unfair or demoralizing to some of the students as students are unique in one or other domain (10).

Secondly, it also affects the uniformity of assessment thus compromising with the quality of examination. Examiners may give more emphasis on final result of examination rather than the ability of student to perform the procedure or examination. There is more thrust on global performance rather than demonstration of individual competencies. Students does not get proper feedback about his performance. Thus, lack of objectivity does not meet the educational objectives and purpose of assessment is not fulfilled (11, 12).

In view of these limitations, there were attempts on incorporating some objectivity in the examination (13, 14). This has resulted in the introduction of Objective structured practical examination (OSPE). OSPE was first conceptualised in form of Objective structured clinical examination (OSCE) and later extended to practical examination in 1979 by Harden and Gleesson (15, 16). OSPE is superior in many ways to conventional pattern of examination. It tries to incorporate all the possible domains of the student assessment by dividing an experiment to various steps and giving weightage at each station. It ensures objectivity in the assessment to the maximum possible extent thus overcoming the flaws in the highly subjective pattern of examination (17). OSPE pattern of examination is utilized in various medical disciplines for formative and summative assessment (18). But this pattern of examination is not also completely flaw-less and amenable to unsafe practices. There is minimum examiner – student interaction so that examiners are now mere observers and just mechanically giving the marks to the students. It has been observed that steps designed in objective check-lists are such that they may not address the important information of a procedure or skill that should be essential for all students to qualify the examination. As this method of
assessment requires large number of examiners, there are chances that many non-medical and untrained staff may be deployed at various stations. This reduces the sanctity of examination and thus may jeopardise the teacher-student relationship. Apart from that, there are many logistics problems and time constraints which make it difficult to implement OSPE. Too much objectivity in the assessment can sometimes lead toinappropriate assessment as breaking complete clinical examination to individual competencies sometimes make it non-meaningful.That has led to incorporation of some about of subjectivity in the OSPE in the form of global score (11, 19). In our study, Linear mixed effects analysis of the relationship between the outcome score and methods of assessment shows the significant effect of types of assessment on the outcome score of the students.

Figure-1 depicts that marks scored in semi-structured format are more in agreement with each other than with conventional assessment. It is also reflected in values of correlation coefficient between the marks given by different examiners (0.754 versus 0.488& 0.466).

It can also be inferenced from figure-2 that there is tendency for giving marks in narrow range to all the students in conventional assessment. Thus, semi-structured tool of assessment appears to be more reliable than conventional tool of assessment.

In terms of acceptability, semi- structured format was more acceptable to students as it provides an opportunity to the student that if he or she is weak in one domain, performance in one or other domain can compensate for that. This approach also addresses the attitude and communication skills of the students.These are very important in the real￾world scenario when students will have to interact with the staff, patients and their attendants once they qualify the final examination. It also ensures the evaluation of student for all the important aspects of the examination related to the experiment and its applied importance and thus minimizing the chances of highly subjective type of assessment. At the same time, this method of assessment is also feasible in a sense that it does not require additional number of examiners and can be accepted and adopted effectively in available amount of resources.

A complete crossed design between examiners and different methods of assessment could be a better study design for comparing the two methods but that may not be feasible in this context.

We suggest utilizing the merits of both the subjective and objective pattern of examination and thus adopting a middle way of assessment so that it will be more convenient for adoption in majority of the medical schools keeping in view of their resources. Medical council of India has already relaxed the norms for faculty and staff requirements for medical colleges to all time low (20). This has resulted in drastic reduction of faculties in various medical colleges to such an extent that OSPE pattern of examination is practically impossible in them. This leads to non-uniformity in assessment of students across the country.

The semi-structured pattern of assessment should be tested and refined further to devise a method of assessment that does not compromise with the basic modalities of assessment tool like validity, reliability, feasibility and acceptability. It should also be convenient to all the personnel related to student assessment so that it can be widely adopted in all the medical colleges across the country.